Systems and methods for analyzing learner&#39;s roles and performance and for intelligently adapting the delivery of education

ABSTRACT

A computer-aided educational system and method to further a student&#39;s understanding of a subject matter through analyzing data captured in an electronic learning system so as to determine correlation data corresponding to variables or trends which are determined to enhance, optimize, and/or improve one&#39;s learning abilities or understanding of educational content. The system and method generates reports based on the correlation data, develops statistical models that highlight learning and behavioral trends, and/or provides recommendations for adapting the learning system based on the correlation data and statistical models.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/058,412, filed Aug. 8, 2018, which is a continuation of U.S. patentapplication Ser. No. 13/180,612, filed Jul. 12, 2011, which claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/363,605 filedJul. 12, 2010. The entire contents of each of U.S. patent applicationSer. No. 16/058,412, U.S. patent application Ser. No. 13/180,612 andU.S. Provisional Patent Application No. 61/363,605 is herebyincorporated by reference, for all purposes.

FIELD

The embodiments herein relate to the field of electronic learning, andin particular to systems and methods for analyzing information relatingto an organization and its members so as to report on various trends ormeasurables. In addition, the systems and methods adapt the delivery oforganizational programs or education based on the analyzed trends andmeasureables so as to enhance the effectiveness and/or efficiency of theeducational delivery.

Introduction

Electronic learning (also called e-Learning or eLearning) generallyrefers to education or learning where users engage in education relatedactivities using computers and other computer devices. For examples,users may enroll or participate in a course or program of study offeredby an educational institution (e.g. a college, university or gradeschool) through a web interface that is accessible over the Internet.Similarly, users may receive assignments electronically, participate ingroup work and projects by collaborating online, and be graded based onassignments and examinations that are submitted using an electronicdropbox.

Electronic learning is not limited to use by educational institutions,however, and may also be used in governments or in corporateenvironments. For example, employees at a regional branch office of aparticular company may use electronic learning to participate in atraining course offered by their company's head office without everphysically leaving the branch office.

Electronic learning can also be an individual activity with noinstitution driving the learning. For example, individuals mayparticipate in self-directed study (e.g. studying an electronic textbookor watching a recorded or live webcast of a lecture) that is notassociated with a particular institution or organization.

Electronic learning often occurs without any face-to-face interactionbetween the users in the educational community. Accordingly, electroniclearning overcomes some of the geographic limitations associated withmore traditional learning methods, and may eliminate or greatly reducetravel and relocation requirements imposed on users of educationalservices.

Furthermore, because course materials can be offered and consumedelectronically, there are fewer physical restrictions on learning. Forexample, the number of students that can be enrolled in a particularcourse may be practically limitless, as there may be no requirement forphysical facilities to house the students during lectures. Furthermore,learning materials (e.g. handouts, textbooks, etc.) may be provided inelectronic formats so that they can be reproduced for a virtuallyunlimited number of students. Finally, lectures may be recorded andaccessed at varying times (e.g. at different times that are convenientfor different users), thus accommodating users with varying schedules,and allowing users to be enrolled in multiple courses that might have ascheduling conflict when offered using traditional techniques.

Despite the effectiveness of electronic learning systems, some users ofan electronic learning system are unable to perform as well as theirpeers. Electronic learning systems have heretofore been unable todetermine the factors associated with some users poor performance, andif current educational delivery mechanisms are ineffective for suchusers, electronic learning systems are unable to alter the deliverymechanism on a student by student basis. In addition, administratorsoften wish to analyze and report on an organization's effectivenesseither for their own purposes or to satisfy requirements set forth byrelevant governing bodies.

Accordingly, the inventors have identified a need for systems, methods,and apparatuses that attempt to address at least some of theabove-identified challenges.

SUMMARY OF VARIOUS EMBODIMENTS

Electronic learning systems may help facilitate the capture ofinformation or data relevant to the learning environment, theorganization, and various stakeholders in the organization (e.g.,administrators, instructors, students, etc.). Information relevant tothese parties may be, for example, historical usage data, performance ofvarious stakeholders, and demographic profiles of the organization andits various stakeholders. Electronic learning systems can thus store andcapture organizational and stakeholder profiles which may be carriedwith the respective stakeholder throughout that stakeholder's career orlife.

Electronic learning systems may be made more effective by analyzing atleast a part of the aggregate captured data. Analysis of such data mayreveal, for example, trends or measureables meeting threshold targets.Because trends and measurables may be used in predictive models, theanalysis of the aggregate captured data has value in that organizationsmay integrate workflows or systems which react (or act in concert with)to predictive modeling.

For example, data which may be captured and further analyzed for trendsand measureables may include: user profile information (e.g., age, sex,name, interests, education, career, etc.), demographic information,learning styles, learning goals, preferred systems, information relatingto devices owned by users and the technical capabilities of suchdevices, accessibility information, past history of users, individualprofiles with whom the user has worked well in the past, enrollmenthistory, withdrawal history, history of achievements, history of usageinformation, keywords used, areas of interest to specific users.

Analysis of at least a part of the information provided above mayproduce aggregate data that highlights historical information, trendinginformation, and measurable information. Such information may be used orharnessed on an organizational or individual level or scope. Theinformation may also have value to subsets of the organization orcertain groupings of individual users within an organization.

In some embodiments, in response to the analyzed data, workflows may bedefined in the educational environment (e.g., within the electroniclearning system). The workflows may use key indicators from thestakeholders involved and may tailor learning experiences to eachstakeholders skill level, and learning or teaching style. For example,in some embodiments, agents, engines, or applications may listen forchanges to key thresholds and initiate additional workflows forremediation, access to advanced materials, and assignment of mentorshipor peer interaction.

For example, when a user begins a new activity such as writing a blogposting, the system may analyze the user's past work and providesuggestions on themes, templates, and other starting materials in orderto assist the user in directing or creating new work. Another example isthat the system may analyze the keywords and materials used by a user(e.g., a student) and suggest additional readings from a variety ofother sources, in addition to sources outside the establishedcurriculum. For example, the additional readings may provide learningmaterial or perspectives that are of different viewpoints than thedefined or established curriculum or of the user's past work. The systemmay also provide the system with the ability to discuss, critique, orsynthesize ideas.

The benefits associated with analysis of the aggregate data captured byelectronic learning systems is not limited to students and targeteddelivery of education to those students. Electronic learning systems mayuse the analysis of such data to provide administrators and instructorswith key indicators for courses and student performance. Administratorsmay use the captured data and analyzed information to report toaccreditation bodies or other stakeholders interested in education orperformance of students. Administrators and instructors may also beprovided with the ability to drill-down into specific students oractivities. The electronic learning system may provide a near real-timedata analysis in a manner which does not affect the performance of theelectronic learning system. The system may classify and quantify studentperformance and further allow manual or automatic flagging andidentifying of behaviors based on student specific, demographic, oraggregated data.

In some embodiments, the analytic engine performing the analysis on thedata may highlight statistically significant data, trends, and helpidentify models which reflect such trend and data. These trends andmodels may then be used by the analytic and/or predictive engines topredict certain events. For example, such predictions may relate to howcertain students will perform in a specific course, or subject matter,and how such students will perform in courses taught by specificteachers. The analytic and/or predictive engines may map a student'sbehavioral model or characteristics onto the courses, instructors,subject matter, curriculum, etc. to enable the electronic learningsystem to determine how successful a specific or type of student may bewith respect to specific courses, curriculum, and/or subject matter. Forthose students which the electronic learning system (e.g., the analyticand predictive engines, specifically) has targeted as at-risk students,some embodiments may include an adaptive engine which adapts at leastone of the delivery mechanisms of the education material, theinstructional method, the educational content, and etc. to attempt todirect the targeted at-risk students towards behavioral characteristicsor academic characteristics that more accurately reflect those ofsuccessful students.

For example, in some embodiments, the analysis of such data may revealsignificant course pattern groupings. For example, the analysis mayillustrate that a positive correlation between the length of time astudent uses a particular tool provided in the electronic learningsystem and grade achieved in the subject matter relating to such usage.Accordingly, for targeted at-risk students, an analytic orrecommendation engine may recommend that the targeted at-risk studentsspend more time using those tools having a positive correlation betweentime spend using such tools and academic success in that subject matter.

In some other embodiments, the analysis may occur post course—that is ananalytics engine may analyze the students success or failures inperformance of a course in the context of a variety of factors. Base onsuch analysis, the analytic engine and/or the predictive engine mayhighlight or recommend courses to students or administrators based onthe analysis that students who tend to like specific courses, may alsotend to like a certain set of other courses.

In some other embodiments, the analytic engine and/or the predictiveengine may highlight or recommend courses taught by professors who theanalytic and/or predictive engine have identified as having a teachingstyle that specific students may find appealing or with which specificstudents may thrive.

In some other embodiments, the analytic engine and/or the predictiveengine may identify anomalies. In some embodiments, such identifiedanomalies may be used to target academic dishonesty. In someembodiments, such identified anomalies may identify students who couldhave done better or could have improved.

In some embodiments, the system may analyze course survey data and/orother relevant data or evidence collected from the e-learningenvironment to make educational recommendations. For example, based onthe analytics of at least a part of the aggregate data, the workflows orsystems may automatically alter or create educational recommendations orthe delivery mechanisms associated with e-learning. An example of suchworkflows or systems may be a recommendation system that optimizes orenhances the match among any combination of at least one student, atleast one course, and at least one instructor. The workflows or systemsmay enable students and/or instructors to understand the respectiveteaching and/or learning patterns. The workflows or systems may providea stakeholder with an understanding of how best to accomplish specifiededucational goals.

In some embodiments, the analytics subsystem or system integrated withthe electronic learning system captures data from various sources ofevidence including, but not limited to, survey data, course assessmentdetails, and usage patterns. This data may then be analyzed to modellearning and teaching patterns. The analytics subsystem or engine maymake recommendations such as identifying correlations between coursepreferences, instructor preferences, performance on quizzes in thecontext of performance in the course, etc.

Based on pattern analysis of at least part of the aggregate captureddata, the electronic learning system may have workflows or subsystemsthat identify stakeholders having similar behaviors and/or similarresultant grades. The stakeholders may be grouped together based onthese behavioral characteristics or other identified variables of thepattern analysis. Thus, based on analysis of the captured data, theelectronic learning system or analytics subsystem or engine may be ableto identify or predict which behavior grouping each student appearstrending towards. The predictive engines may predict or recommend how astudent can improve from the predicted behavioral grouping to the nexthigher ranking of behavior. In other words, the predictive engine mayhelp stakeholders identify trends, and more specifically, to identifypotentially problematic students (e.g., at-risk students) and recommenda course of action as to how to change course for the grouping for whichthe identified problematic student is heading towards. Such recommendedcourse of actions may include, for example, recommended readings,recommended homework, recommended discussion tools, recommendedpartners, etc. Alternatively, the analytics engine (or subsystem) mayidentify behaviors or learning delivery mechanism which correspond topositive factors—that is the analytics engine may identify those factorswhich appear to enhance the ability for students to comprehend and learnthe educational curriculum. By identifying the factors positivelycorrelated with good performance, the system may synthesize aeducational delivery mechanism or curriculum that is specific to eachstudent in attempt to personalize the educational experience in such away that enhances or optimizes each respective student's learningexperience.

In some embodiments, the electronic learning system may include ananalytic engine (or subsystem) and a predictive engine (or subsystem).The analytic engine and the predictive engine may be separate anddistinct engines, or they may be fully integrated and thus notidentifiable from one another. The analytics engine may track usagepatterns of the e-learning environment (e.g., usage for a course). Basedon the analyzed information, the predictive engine may combineidentified patterns with additional and diverse evidence collected overtime in order to predict a stakeholder's (e.g., a student's or aninstructor's) performance with a measured degree of confidence.

The combination of an analytics engine and a predictive engine empowersstakeholders (e.g., instructors and students) to understand how aneducational environment (e.g., a course environment) is utilized and toidentify and understand the correlation between the usage data and thelevels of achievement of stakeholders (e.g., students).

According to one embodiment, there is provided an electronic learningsystem, comprising: a plurality of computing devices for communicatingwith a plurality of users in an educational community; at least oneserver in communication with each of the plurality of computing devices,at least one of the servers being in communication with at least onedata storage device configured to store information associated with atleast one of organizational data, and usage data, and at least one ofthe servers being in communication with at least one storage deviceconfigured to host at least one analytics engine, wherein the analyticsengine is configured so as to analyze the usage data and generatereports on statistical trends or measurables.

According to one embodiment, there is provided an electronic learningsystem, comprising: a plurality of computing devices for communicatingwith a plurality of users in an educational community; at least oneserver in communication with each of the plurality of computing devices,at least one of the servers being in communication with at least onedata storage device configured to store information associated with atleast one of e-learning environment data, organizational data, and usagedata, and at least one of the servers being in communication with atleast one storage device configured to host at least one analyticsengine, wherein the analytics engine is configured so as to analyze atleast one of the organizational data and the usage data and based onanalysis, generate at least one recommendation for adapting a learningenvironment.

In some embodiments, the analytics engine may analyze information fromthe e-learning environment (e.g., e-learning environment data,organizational data, and usage data) and make recommendationscorresponding to the matching among students, among courses, amonginstructors, or among one or more of students, courses and instructors.The at least one recommendation allows stakeholders in an organization(e.g., administrators, students, instructors, etc.) to better understandteaching and learning patterns and how educational goals can beachieved. This information may then be applied to adapt learningenvironments to enhance the effectiveness of the educational deliverymechanisms. For example, the analytics engine may analyze at least oneof learning environment data, organizational data, or usage data (e.g.,user performance, user trends, historical data) and/or report oninstructor trends, student trends, behavioral trends of users orcorrelation data corresponding to instructor preferences (e.g.,preferred educational delivery mechanisms, and preferences for groupingstudents or learning topics), and student preferences (e.g., types ofcourses, educational offerings by certain professors, educationalofferings certain types of professors, educational offerings beingdelivered by way of particular mechanisms, and course and/or subjectmatter preferences).

In some embodiments, the analytics engine may analyze a stakeholdersurvey and further analyze a subset of the stakeholder survey in orderto present patterns or trends in the form of a mosaic plot, which helpreaders visualize multi-dimensional contingency tables. The subset ofthe survey may be manually chosen by the individual requesting thereport. Alternatively, a report generating engine, which may be a subsetof the analytics engine, may automatically select the subset of thesurvey based on the answers submitted for the survey. Specifically, thereport generating engine may select those questions having a set ofanswers that meet some threshold level of certainty of the expressedopinion or knowledge. For example, the analytics engine may analyzeperformance trends on a quiz or survey. Based on this analysis, a reportmay be generated to communicate performance trends that illustratecorrelation between performance on a certain question or set ofquestions with performance on a different question or set of questions.Specifically, the report may illustrate that users (e.g., students) whoanswered a specific question (or set of questions) incorrectly alsotended to answer another question (or set of questions) incorrectly. Inother words, the analytics engine may determine positive correlationsbetween performances on a first set of questions with performance onsecond set of questions. Alternatively, the analytics engine maydetermine negative correlations between performances on a first set ofquestions with performance on a second set of questions.

In some embodiments, reports may be generated so as to illustraterelevant information in the form of at least one of: a Mosaic plot, aheat diagram, a correlogram, a pie chart, a tree diagram, and a chart.In some embodiments, colors may be used to identify strongercorrelation. For example, a dark blue color may identify strong positivecorrelation among variables. A lighter blue color may identify a weakerpositive correlation among variables. A dark red color may be used toidentify a strong negative correlation among variables, and a light redcolor may be used to identify a weaker negative correlation amongvariables.

In some embodiments, a color and percentage of a pie graph may identifythe correlation among variables. For example, a blue correlation maycorrespond to a positive correlation among variables. The greater theextent to which the pie or shape is shaded blue, the stronger thepositive correlation may be. Similarly, a red correlation may correspondto a negative correlation among variables. The greater extent to whichthe pie or shape is shaded red, the stronger the negative correlationmay be.

In some embodiments, there is a method for analyzing informationcaptured in an electronic learning system, the method comprising:identifying a plurality of users in an educational community; providinga plurality of computing devices for communicating with the plurality ofusers in the educational community; providing at least one server incommunication with each of the plurality of computing devices, eachserver having at least one data storage devices coupled thereto andconfigured to store information associated with at least one ofe-learning environment data, organizational data, and usage data, and atleast one server being configured to host an analytics engine, where inthe analytics engine is configured to analyze the at least one ofe-learning environment data, organizational data, and usage data.

In some embodiments, the analytics engine is further configured togenerate at least one report on at least one of: at least onestatistical trend and at least one measurable.

In some embodiments, the analytics engine analyzes data and determinescorrelation data that corresponds to an interaction between at least twovariables of an electronic learning system.

In some embodiments, the analytics engine is further configured togenerate at least one recommendation for adapting a learning environmentpresented to at least one of the plurality of users.

In some embodiments, based on the correlation data, the analytics enginegenerates at least one recommendation corresponding to mechanisms forenhancing at least one of the plurality of user's interaction with anelectronic learning system.

In some embodiments, the electronic learning system and/or electronicportfolios corresponding to each stakeholder, may store personalidentification information. Such personal identification information mayinclude, but is not limited to, a password, a login name, a stored image(e.g., of the facial characteristics, among other identifying features),a finger print, a voice sample, etc.

In some embodiments, there may be at least one of video, photographic,and audio capturing devices in all classrooms or in locations spreadthroughout an organization. Such devices may configured and integratedto the electronic learning system in such a way that the devices capturea sample and the electronic learning system compares that sample to thestored data on each stakeholder. Upon comparing and matching the samplewith the stored data to a predetermined or specific threshold whichcorresponds to a positive match, the electronic learning system may logor otherwise keep track of the presence of such stakeholders. In someembodiments, this recognition of a stakeholders presence throughout theorganization may automatically take attendance for a course or for aprogram. In some embodiments, a workflow may be generated that informsthe stakeholders of any absences. In some embodiments, such stakeholdersmay be allowed to argue against the absence. In some embodiments,remedial education or other efforts may be directed to the stakeholdersafter such stakeholders are recorded to have been absent more than athreshold number of times.

In some embodiments, the electronic learning system may work with ananalytic and/or predictive engine to arrange a seating arrangement(e.g., in a classroom, in a lab, etc.) based on the behaviorcharacteristics and/or preferences of the stakeholders. For example, thesystem may determine or predict what the best (e.g., or more favorable)seating arrangement to facilitate an interactive classroom discussionbased on personal preferences, behavior characteristics, andunderstanding of the subject matter. In some embodiments, the system maygroup stakeholders for certain projects (e.g., study projects, labs,etc.). The system may group these stakeholders in such a way tofacilitate learning. Such grouping may be based on the respectivestakeholders strengths and weaknesses.

DRAWINGS

For a better understanding of the embodiments described herein and toshow more clearly how they may be carried into effect, reference willnow be made, by way of example only, to the accompanying drawings whichshow at least one exemplary embodiment, and in which:

FIG. 1 is a block diagram that illustrates the interaction betweencomponents in an electronic learning system according to one embodiment.

FIG. 2 is a graph that illustrates historical usage data of anelectronic learning system according to one embodiment.

FIG. 3 is a chart that illustrates quiz statistics according to oneembodiment.

FIG. 4 is a chart that illustrates a detailed quiz statistics accordingto one embodiment.

FIG. 5 is a chart that illustrates the use of a topic delivered on anelectronic learning system according to one embodiment.

FIG. 6 is a screenshot that illustrates a report query page according toone embodiment.

FIG. 7 is a screenshot that illustrates a report data selection pageaccording to one embodiment.

FIG. 8 is a screenshot that illustrates the report setup page accordingto one embodiment.

FIG. 9 is a screenshot of the report setup page according to oneembodiment.

FIG. 10 is illustrates a screenshot of the report preview page accordingto one embodiment.

FIG. 11 illustrates a report according to one embodiment.

DESCRIPTION OF VARIOUS EMBODIMENTS

It will be appreciated that numerous specific details are set forth inorder to provide a thorough understanding of the exemplary embodimentsdescribed herein. However, it will be understood by those of ordinaryskill in the art that the embodiments described herein may be practicedwithout these specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not toobscure the embodiments described herein.

Furthermore, this description is not to be considered as limiting thescope of the embodiments described herein in any way, but rather asmerely describing the implementation of the various embodimentsdescribed herein.

In some cases, the embodiments of the systems and methods describedherein may be implemented in hardware or software, or a combination ofboth. However, in some cases, these embodiments are implemented incomputer programs executing on programmable computing device eachcomprising at least one processor, a data storage device (includingvolatile and non-volatile memory and/or storage elements), at least oneinput device, and at least one output device.

For example and without limitation, the computing device may be amainframe computer, a server, a personal computer, a laptop, a personaldata assistant, a tablet computer, a smartphone, or a cellulartelephone. Program code may be applied to input data to perform thefunctions described herein and generate output information. The outputinformation may be applied to one or more output devices, in knownfashion.

Each program may be implemented in a high level procedural or objectoriented programming and/or scripting language to communicate with acomputer system. However, the programs can be implemented in assembly ormachine language, if desired. In any case, the language may be acompiled or interpreted language. Each such program may be stored on anon-transitory storage media or a device (e.g. ROM or magnetic diskette)readable by a general or special purpose programmable computer, forconfiguring and operating the computer when the storage media or deviceis read by the computer to perform the procedures described herein. Suchprograms and data associated with such, may be stored on data storagedevices. The data storage devices may include volatile or non-volatilecomputer memory such as RAM, flash memory, video memory and magneticcomputer storage devices. The particular storage of various programs orassociated data may be stored on different storage devices and/ordifferent storage mediums. For example, a first storage device thatstores a portion of the program or data to be stored may include aslower hard disk drive (e.g. a persistent data storage device) while asecond data storage device that stores another portion of the program ordata to be stored may include a faster RAM (e.g. a dynamic data storagedevice).

The systems and methods as described herein may also be considered to beimplemented as a non-transitory computer-readable storage medium,configured with a computer program, where the storage medium soconfigured causes a computer to operate in a specific and predefinedmanner to perform at least some of the functions described herein.

Referring to FIG. 1, illustrated therein is an electronic learningsystem 1. The electronic learning system includes at least one server incommunication with a plurality of devices, which in turn communicatewith a plurality of users in an educational community. The plurality ofusers interface with a learning environment 2 and access educationalcontent and delivery mechanisms for such to learn a curriculum or asubject matter. By way of the interaction between the plurality of usersin an educational community and the learning environment, information 3is captured by the server and stored on at least one storage devicesthat is coupled to the at least one server. The information 3 may be,for example, e-learning environment data, organizational data, and usagedata. This data may represent organizational, institutional,departmental data corresponding to structure or educational workproduct. This data may also represent information generated throughhistorical usage. This data may also represent information aboutindividuals such as, for example, various stakeholders includingadministrators, teachers, and students.

The electronic learning system also has a workflow engine 4 (e.g., ananalytics engine) which analyzes the information 3 and generates variousreports 5 or recommendations which relate to statistical information andtrends. This reports also contain or are based on correlation data whichthe workflow engine 4 has identified as being statistically relevant.

In some embodiments, the analytics engine may analyze users historicaldata. For example, the analytics engine may be queried to report on thehistorical access information of a subset of an educationalinstitution's student body. Such an embodiment may be useful foradministrators in their reporting on the institution's compliance withvarious targets and/or regulations. In addition, this information may behelpful in identifying students prone to academic problems, or thosestudents falling outside the behavior norms associated with successfulstudents. If administrators and/or instructors have access to thisinformation quickly and early-on, the administrators and/or instructorsmay be able to help guide those students needing guidance, or theadministrators and/or instructors may be able to adapt the teachingdelivery to a mechanism that invokes the interests of these identifiedstudents.

The reporting characteristics of the analytics engine may allowadministrators and/or instructors to examine the differences in howdifferent students use the electronic learning system and the courseresources that the instructor and/or institution has created. Theadministrator and/or examiner may also examine how the usage changesthrough a specified time period or how the usage of certain tools may bedistinguished from that of other tools. These reporting mechanisms allowfor the discovery of clusters as they relate to the usage of theelectronic learning system and resources created therein.

FIG. 2 illustrates such a report generated by the analytics engine. Thereport 20 illustrates historical data corresponding to a recordation ofthe number of times a course site has been accessed on the electroniclearning system. The report 20 includes a list of the user identifiers21, a list of the dates for which the report was queried 22 and numbers23 identifying the number of times that a user has accessed the courseon a respective date.

In some embodiments, the analytics engine may analyze course data todetermine statistically relevant trends or information. For example, theanalytics engine may analyze the results of a quiz or test. Based onsuch an analysis, the analytics engine may determine or identifyquestions which were too easy, too difficult, or ambiguous. Thisanalysis relies on determining whether a statistically relevant numberof users similarly answered questions on a quiz. An instructor mayreceive a report or analysis of the quiz or test results and learn thatcertain subject matter was not fully grasped by a statistically relevantnumber of students, or that a question may have been too difficult, ordifficult to understand because a statistically significant number ofstudents failed to answer the question correctly. The analysis enginemay also highlight academic integrity concerns if some students answeredthe questions in a way that was significantly relevant and/or indicatedthat the students may have communicated or aligned the responses to thequiz. For example, the analytics engine may analyze when a quiz wassubmitted, and determine whether a group of students submitted a quizsubstantially at the same time, and that the responses to the submittedquizzes were substantially similar to the point where the analyticsengine determined that the correlation between the submitted quizzes wasstatistically relevant and that concerns of academic integrity may beinvoked.

In some embodiments, the analytics engine may analyze course data suchthat statistically relevant trends or information assist in highlightingwhat pattern of usage is significant in terms of a correlation betweenthe usage and a user's grades. This analysis may assist administratorsand instructors better understand the picture (e.g., the behavioralcharacteristics and preferences) of a successful student. Such ananalysis may help predict students prone to withdrawing from coursesearly. The analysis may help predict enrollment trends, particularly asthey relate to departments, curriculum, and/or courses. This informationmay be helpful to administrators in allocating resources to departments,curricula, and/or courses in high enrollment demand.

In some embodiments, the reporting mechanisms of the analytics enginemay allow administrators and/or instructors to model the relationshipbetween grades and tool access patterns. In other words, theadministrators and/or instructors may be able to analyze theeffectiveness of certain learning tools, and the ineffectiveness ofcertain other learning tools. Further, administrators and/or instructorsmay harness the analytics engine to explore the most significantfactor(s) that contribute to student success within a course, acurriculum, and/or subject matter. The administrators and/or instructorsmay find value in analyzing these patterns or trends both at the end ofa course′ delivery or during the course. Based on this information,administrators and/or instructors may be armed with data that allowsthem to adapt future course offerings, or to assist other instructors increating electronic learning curriculum and materials associatedtherewith.

For example, FIG. 3 illustrates a chart 30 that displays statisticalinformation relating to responses to quiz questions. The chart mayprovide background information as to the question name 31 and type ofanswer called for 32. The chart may also illustrate correlation data 33and a distribution as to the question score and final quiz scoredistribution 34. The analytics engine may query and display thisinformation in a near real-time mechanism. Thus, a teacher oradministrator may quickly access performance information which allowthose stakeholders to gauge the effectiveness of the teaching, teachingmechanisms, and/or educational content.

FIG. 4 illustrates an expanded chart that displays statisticalinformation relating to responses to quiz questions. The chart 40 ofFIG. 4 is more detailed than the chart 30 of FIG. 30, because itdisplays a break-down 45 of the percentage of students which provided asimilar answer.

In some embodiments, the analytics engine may be utilized to provideadministrators with detailed information that corresponds to thepopularity of certain courses, subject matter, or curricula. Inparticular, the analytics engine may provide a outline the number ofhits that a course, subject matter, module, or topic had over the courseof a defined period of time. Such information may communicate toadministrators the extent to which the electronic learning system isbeing used by students, instructors, and/or departments.

FIG. 5 illustrates a report 50 which communicates the number of hits 51a course or topic 52 had over a predetermined period of time. Inaddition, FIG. 5 illustrates the throughput 53 or use that such a courseor topic 52 experienced over the predetermined period of time.

In some embodiments, the administrators and/or instructors may have alarge degree of control over the type of reporting done by the analyticsengine. For example, FIG. 6 illustrates an interface 60 which theadministrators and/or instructors may use in submitting report queries.For example, the administrators and/or instructors may choose the typeof domain 61 for which their reports will be generated. The domains fromwhich the administrators and/or instructors may choose to query mayinclude, but are not limited to, client access (e.g., historical accessinformation, etc), content access (e.g., historical access information,etc), enrollments, grades, IIS monthly stats, Org Unit Access (e.g.,historical access information, etc), quiz questions, quiz summaries,sessions, and tool access (e.g., historical access information, etc).

In some embodiments, the analytics engine may also allow users (e.g.,administrators, instructors, and/or students), to select the format inwhich the queried report is displayed. The format may be selected in thereport format window 62 illustrated in FIG. 6. The report may begenerated in formats including, but not limited to, tables, charts, andcrosstabs displays.

In some embodiments, the analytics engine may be widely configurable tothe needs of stakeholders (e.g., administrators, instructors, and/orstudents). For example, the analytics engine may allow a stakeholder tospecifically identify data sets which are analyzed by the analyticsengine and for which reports are generated. FIG. 7 illustrates ascreenshot of the report data selection page 70 in which stakeholdersmay select the data 71 to be included in the report. Further, asillustrated in FIG. 8, the stakeholders may configure the analyticsengine by specifying or at least having some input into the format ofthe report.

FIG. 8 illustrates a screenshot of the report setup page 80. The reportsetup page allows the stakeholders to specify which data to include inthe report, the characteristics of this data (i.e., what role the datawill have in the report) and to specify the identifiers in the report.Specifically, the stakeholder can specify the characteristics of thedata, or how the data will be used in the report, by selecting thecharacteristics from the characteristics window 81. In addition, thestakeholder may specify identifiers such as the title on the report byinputting the identifier into the identifier window 82.

FIG. 9 illustrates a screenshot of the report setup page 90 in whichstakeholders may further define the characteristics of the report to begenerated by the analytics engine. For example, a stakeholder may selectthe units in which the queried data is displayed upon generation of thereport. The units may be selected in the data group characteristicswindow 91.

Finally, FIG. 10 illustrates a screenshot of the report preview page100. As seen in FIG. 10, the individual data 101 a, 101 b, 101 c, etc.,may be populated in the report format selected by the stakeholder. Thisparticular previewed report 100 illustrates the user access with respectto dates.

FIG. 11 illustrates a report 110 according to one embodiment. The reportprovides information relating to the user 112 access patterns of tools111 with respect to bandwidth. Stakeholders may use such a report togauge the usefulness and popularity of certain tools.

While the steps of the above methods have been described sequentiallyhereinabove, it should be noted that sequential performance of the stepsmay not need to occur for successful implementation of the method. Aswill be evident to one skilled in the art, rearranging sequence ofperformance of the steps, omitting the performance of some steps, orperforming the steps in parallel may be possible without abandoning theessence of the invention.

While certain features have been illustrated and described herein, manymodifications, substitutions, changes, and equivalents will now occur tothose of ordinary skill in the art. It is, therefore, to be understoodthat the appended claims are intended to cover all such modificationsand changes as fall within the true spirit of the invention.

1. An electronic learning system comprising: a) a plurality of computingdevices that communicate over a network with a learning managementsystem; b) at least one server configured to: (i) provide the learningmanagement system over the network; (ii) communicate with the pluralityof computing devices; (iii) store information for the learningmanagement system, the information associated with at least one ofe-learning environment data, organizational data and usage data, theinformation comprising aggregate data based on interactions between aplurality of users and the learning management system via the computingdevices; and (iv) implement at least one analytics engine, wherein theanalytics engine is configured to analyze the at least one of thee-learning environment data, the organizational data, and the usagedata, identify usage patterns based at least in part on the aggregatedata, the usage patterns including at least one positive correlationbetween academic performance and an interaction with the learningmanagement system based at least in part on the usage data, to generateat least one report on at least one of statistical trends andmeasureables, and to recommend to at least one of the plurality of usersat least one tool provided by the learning management system based atleast in part on academic performance of the at least one user and thepositive correlation between academic performance and an interactionwith the learning management system.
 2. The system of claim 1, whereinthe at least one positive correlation data corresponds to at least onevariable that enhances an educational experience for at least one of theplurality of users.
 3. The system of claim 2, wherein the at least onepositive correlation data corresponds to factors relating to at leastone of user demographic information, user behavioral characteristics,user learning preferences, user teaching preferences, educationaldelivery mechanisms.
 4. The system of claim 1, wherein the analyticsengine determines at least one negative correlation data within thelearning management system.
 5. The system of claim 4, wherein the atleast one negative correlation data corresponds to at least one variablethat acts as a detriment to an educational experience for at least oneof the plurality of users.
 6. The system of claim 5, wherein the atleast one negative correlation data corresponds to factors relating toat least one of user demographic information, user behavioralcharacteristics, user learning preferences, user teaching preferences,educational delivery mechanisms.
 7. The system of claim 1, wherein theat least one generated report is in the form of at least one of: amosaic plot, a heat diagram, a correlogram, a pie chart, a tree diagram,and a chart.
 8. The system of claim 1, wherein the at least one reportidentifies specific learning users that are performing at a level belowa predetermined threshold.
 9. The system of claim 1, wherein the atleast one report identifies subject matter in an educational curriculumwhich was not adequately understood by learning users.
 10. (canceled)11. The system of claim 1, wherein the at least one report communicatesthe learning delivery mechanism to which at least one specific learninguser responds better than other learning delivery mechanisms.
 12. Acomputer program product for providing an electronic learning system,the computer program product being embodied in a non-transitory tangiblecomputer readable storage medium and comprising computer instructionsfor: providing the learning management system to a plurality ofcomputing devices over the network, wherein to provide the learningmanagement system includes to communicate with the plurality ofcomputing devices; storing information for the learning managementsystem, the information associated with at least one of e-learningenvironment data, organizational data and usage data, the informationcomprising aggregate data based on interactions between a plurality ofusers and the learning management system via the computing devices,implementing at least one analytics engine, wherein the analytics engineis configured to analyze the at least one of the e-learning environmentdata, the organizational data, and the usage data, identify usagepatterns based on the aggregate data, the usage patterns including atleast one positive correlation between academic performance and aninteraction with the learning management system based at least in parton the usage data, to generate at least one report on at least one ofstatistical trends and measureables, and to recommend to at least one ofthe plurality of users at least one tool provided by the learningmanagement system based at least in part on academic performance of theat least one user and the positive correlation between academicperformance and an interaction with the learning management system. 13.The computer program product of claim 12, wherein the at least onepositive correlation data that corresponds to at least one variable thatenhances an educational experience for at least one of the plurality ofusers.
 14. The computer program product of claim 12, wherein the atleast one positive correlation data corresponds to factors relating toat least one of user demographic information, user behavioralcharacteristics, user learning preferences, user teaching preferences,educational delivery mechanisms.
 15. The computer program product ofclaim 12, wherein the analytics engine determines at least one negativecorrelation data within the learning management system.
 16. The computerprogram product of claim 15, wherein the at least one negativecorrelation data corresponds to at least one variable that acts as adetriment to an educational experience for at least one of the pluralityof users.
 17. The computer program product of claim 16, wherein the atleast one negative correlation data corresponds to factors relating toat least one of user demographic information, user behavioralcharacteristics, user learning preferences, user teaching preferences,educational delivery mechanisms. 18.-29. (canceled)
 30. A method foranalyzing information captured in an electronic learning system, themethod comprising: a) identifying a plurality of users associated with alearning management system; b) providing, over a network, the learningmanagement system to a plurality of computing devices associated withthe plurality of users associated with the learning management system;c) storing information associated with at least one of e-learningenvironment data, organizational data, and usage data, the informationcomprising aggregate data based on interactions between a plurality ofusers and the learning management system via the computing devices; d)implementing an analytics engine, wherein the analytics engine isconfigured to analyze the at least one of the e-learning environmentdata, the organizational data, and the usage data, identify usagepatterns based at least in part on the aggregate data, the usagepatterns including at least one positive correlation between academicperformance and an interaction with the learning management system basedat least in part on the usage data, to generate at least one report onat least one of statistical trends and measureables, and to recommend toat least one of the plurality of users at least one tool provided by thelearning management system based at least in part on academicperformance of the at least one user and the positive correlationbetween academic performance and an interaction with the learningmanagement system.
 31. The method of claim 30, wherein the analyticsengine analyzes data and determines correlation data that corresponds toan interaction between at least two variables of an electronic learningsystem.
 32. (canceled)
 33. The method of claim 32, wherein based on thecorrelation data, the analytics engine further recommends mechanisms forenhancing at least one of the plurality of user's interaction withlearning management system.
 34. (canceled)
 35. The system of claim 1,wherein the interactions comprise the times of electronic submissions ofquizzes by the plurality of users and the responses submitted within thequizzes, and wherein the usage pattern identifies a concern regardingacademic integrity in connection with plurality of students thatsubmitted corresponding quizzes based at least in part on whether thequizzes are submitted within a threshold amount of time and whether acorrelation among the responses to the quizzes exceeds a presetcorrelation threshold.